Abstract Title Page 


Title: 


“STEMming” the Swell of Absenteeism in Urban Middle Grade Schools: 
Impacts of a Summer Robotics Program 


Authors and Affiliations: 


Martha Abele Mac Iver and Douglas J. Mac Iver 
Johns Hopkins University 
School of Education 


SREE Fall 2014 Conference Abstract Template 



Abstract Body 


Problem / Background / Context: 

Attendance is probably the most fundamental behavioral indicator of student engagement 
with school. Though many students fall off-track to success for the first time in ninth grade, 
poor attendance patterns often begin increasing in middle school and become worse in high 
school. Recent studies indicate higher rates of chronic absenteeism in grade 8 than in earlier 
middle grades in such states as Oregon and Nebraska (Balfanz & Byrnes, 2012) and alarming 
rates of middle grades absenteeism in urban districts such as New York City and Baltimore (e.g., 
Balfanz & Byrnes, 2013; Mac Iver et ah, 2008). Rigorous research on interventions to improve 
student attendance remains in the early stages. Evaluation studies of out-of-school time (OST) 
programs, both summer school and after school, have found mixed results regarding program 
impacts on school attendance (Dynarski et ah, 2004; Gottfredson et ah, 20 10; Lauver,2002). 
Cooper et al. (2000, p. 102) specifically advocate the need for summer program evaluations to 
investigate such outcomes as attendance. 

Missing school during the secondary grades can often be traced to low levels of 
motivation. As Eccles (2008) has so aptly summarized the crux of the motivation issue, it often 
boils down to two main questions in students’ minds about what happens in school: “Can I do 
the task?” And “Do I want to do the task?” (Eccles & Midgely, 1989; Meece, 2003). Recent 
discussions of noncognitive factors affecting academic performance have emphasized the 
importance of developing an academic mindset to influence academic behaviors such as 
attendance and exerting effort in class and homework assignments (Earrington et ah, 2013). The 
process of helping students to internalize these beliefs can occur not only in the core academic 
classroom, but also in elective activities like robotics that build a sense of competence and value 
in academic pursuits. A number of studies have begun to investigate the impacts of one such 
hands-on activity, robotics, but the research evidence remains rather thin. Benitti’s (2012) 
literature review identified just ten articles that included quantitative measurement of student 
learning. The reviewed studies reported some positive effects on learning of science concepts 
and some positive effects on mathematics learning for some subgroups, but did not address 
noncogitive outcomes such as attendance as a measure of school engagement. 

Purpose / Objective / Research Question / Focus of Research: 

Given the salience of attendance as a predictor of student achievement outcomes, 
additional research on effective means of increasing attendance for at-risk students is particularly 
important. In this paper we focus primarily on the following research question: Did the five 
week STEM robotics summer learning program (described below) have a positive impact on the 
following year’s attendance rate of middle school students (compared to a matched sample of 
students who did not receive any of the district’s summer programs)? 

Improvement Initiative / Intervention / Program / Practice: 

The focus of this study is a STEM Robotics Summer Eeaming Program funded by the 
U.S. Department of Education’s Investing in Innovation (i3) program in a development award. 
The primary goal of this five-week summer program implemented by the school district was to 
provide additional out of school time focused on mathematics instruction and robotics so that 
enrolled students could increase their mathematics grade-level aptitude by the end of the 
program and develop interest in technology and STEM college majors and careers. The robotics 
component was expected to increase student engagement (including attendance) and perception 
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of the relevance of mathematics, leading to increased student effort and math achievement. 

Setting: 

The program was developed and conducted in 8 school sites citywide by an urban school 
district with 85% of students eligible for free/reduced price lunch and 92% African-American or 
Hispanic. 

Population / Participants / Subjects: 

All district students in grades 5 to 7 (rising 6**'-8* graders) were eligible to enroll in the 
program. The program specifically targeted students who were low-performers in mathematics 
on the previous year’s state assessment. In Summer 2012, a total of 193 students within the 
specified grade levels were enrolled in the summer robotics program in eight different sites 
throughout the city. A total 166 students were enrolled in 2012-13 in grades 6 to 8 in the district, 
had not been retained in grade, and had both a 2013 and 2013 mathematics score. Table 1 
summarizes the significant demographic and behavioral differences between the students who 
attended the summer school robotics program from the full group of students who did not attend 
summer school (as evidenced especially in their attendance rates during the regular school year 
and the large majority of males as program participants), as well as differences between the prior 
characteristics of their post-program schools. 

Research Design: 

Program pre-enrollment and district policy did not permit random assignment of students 
to the summer program treatment condition. The research design was quasi-experimental, with 
program students matched to comparison students who did not attend summer school using 
propensity score and Mahalanobis metric matching. Only potential control students who did not 
attend summer school and who did have data on the prognostic covariates (prior year’s 
attendance and prior year’s state mathematics z-score) as well as the outcome variables were 
included in the matching analyses. In this two-step method, all control subjects meeting these 
requirements who were within ± .2 of the estimated propensity score of each treated subject were 
identified as potential matches. Then, Mahalanobis metric matching on the two prognostic 
covariates was used to make a final selection of up to three matches for each treated student. 
Within each prior grade level (fifth, sixth, and seventh), we selected a comparison group 
subsample from among our larger sample of potential control subjects (so that the comparison 
group subsample had similar covariate values to the treatment sample on 15 covariates, including 
on the 2 key “prognostic” covariates). All of the matching was performed using nearest- 
remaining-neighbor matching, beginning with the most difficult to match treated subject (the one 
with the highest propensity score) and proceeding to the subject with the lowest propensity score. 
The propensity scores were estimated using logistic regression with linear terms for each 
covariate. 

Data Collection and Analysis: 

The district shared program record data and student administrative data with the research 
team. The primary outcome variable for this study was yearly attendance rate (percent of days 
attended), calculated for all students from district administrative records on attendance in the 
year following the summer program treatment. Mathematics achievement in the spring 
following the summer program intervention was another outcome variable for the overall study. 
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The treatment variable indieates for eaeh student whether he or she was in the Robotics 
enrichment during summer school group (coded 1) or the matched comparison group with no 
summer school (coded 0). All covariates were grand mean centered in the impact models. 
Students were nested in the summer treatment sites, not the post-treatment schools, so in addition 
to their individual characteristics we estimated as level 1 variables the value of school level 
characteristics (measured the year prior to intervention) of the school attended the year following 
the intervention. All covariates were pre-specified and included in the final model, regardless 
of their statistical significance (see list in Table 4). 

We used a two-level fixed effects model with covariates that assumed homogeneity of the 
treatment effects across sites. The treatment students were nested in eight summer treatment 
sites, and control students were nested together in a ninth site (no treatment). This follows the 
constant block effect model described by Dong and Maynard (in press). 

Level 1 describes the relationship between students’ outcomes, student-level characteristics, and 
their treatment status. The level 1 model is 

Yij = Poj + PijTi + SPisXsij + Cij, 

where 

Lyis an outcome for student / in sitcy; 

Ti is 1 if the student is the treatment group and 0 otherwise; 

Xij is a set of S student-level covariates (described above) for student / in site y, measured 
in the year prior to treatment exposure and centered on the grand mean in the sample; and 
e,y is a random error term for student / from site y, assumed to be independently and 
identically distributed across students within sites (i.e., the “within-site” residual). 

Level 2: Sites 

f^oj = Too + Uq 
Pij = rio 

p2s = Y2s (and so on for each covariate) 

where 

Yoo is the grand mean 

Yio is the main effect of treatment 

The set of Y 2 s regression coefficients represent the relationships between students’ 
outcomes and the covariates, with each coefficient assumed to be constant across sites, 

Uoj J=l,. . .,J are fixed effects associated with each site effect, and are constrained to have a mean 
of zero. All available covariates described earlier were included in the final model, regardless of 
their statistical significance. To test for baseline equivalence between the treatment and control 
students on attendance, we estimated a hierarchical linear model in the form specified above in 
which prior year’s attendance was predicted by treatment status (controlling for grade level 
dummy variables). 

Findings / Outcomes: 

Baseline equivalence between the treatment and control group was achieved (Table 3). 

As shown in Table 4, the adjusted mean attendance rate of the treatment students was 1.4 
percentage points higher than control students. This impact was both statistically significant (t 
(631) = 3.52, p =.001), and large enough to be educationally meaningful. Glass’s A = .34 
Another way of stating the impact is that treatment students attended about 2.5 days more of the 
1 80-day school year on average. Parallel analyses were conducted on the subgroup of low- 
achieving students (60 treatment students who scored Basic on the math pre-test and their 167 
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matches from the eomparison group). Baseline equivalenee was again aehieved. As shown in 
Table 4, the adjusted mean attendanee rate of the treatment students in the subsample was 2.6 
pereentage points higher. This impaet was both statistieally signifioant (t (206) = 2.865, p. = 
.005), and large enough to be educationally meaningful. Glass’s A = .33. Another way of stating 
the impact is that treatment students in the subsample attended, on average, about a week more 
of sehool than did the eontrol students in the subsample (i.e., attended 4.7 days more during the 
eourse of the 180-day sehool year.) 

Conclusions: 

It is important to point out the limitations of this study. Although the propensity seore 
matehing yielded a eomparison group with baseline equivalenee on prior attendanee, and the 
analysis also eontrolled for all covariates, the study’s quasi-experimental design, like all sueh 
designs, was subjeet to potential unmeasured bias in the self-seleetion of summer program 
partieipants. It is possible that even though matehed comparison groups students were 
equivalent to program students on prior sehool attendanee rates, as well as on all the other 
matching variables, they differed in some unmeasured way that would explain their higher 
sehool attendanee in the year following the program. At the same time, while random 
assignment of students to the summer sehool program or eontrol eondition would have been a 
stronger design, sueh a design would have been subjeet to non-eomplianee among the students 
assigned to summer sehool treatment and to differential attrition that eould undermine internal 
validity. 

Despite its limitations, the findings of this study emphasize the importanee of 
investigating the potential impaet of out-of -sehool programs on sehool-foeused engagement. As 
Lawson and Lawson (2013) argue, researeh on sehool engagement needs to move beyond the 
traditional classroom and school to include out-of-sehool and community-foeused aetivities. 
Aetivities outside of the regular sehool sehedule ean potentially build developmental 
eompetencies — particularly feelings of eonfidenee, competenee, and eonneetion — that can keep 
students attaehed enough to sehool through attendance to inerease their likelihood of suceess 
after leaving high sehool. Further researeh on the impaet of similar programs on student 
engagement measures (in addition to aeademie aehievement measures) will be a useful 
investment. 

“Motivating the academieally unmotivated “ is one of the critical issues of the 2L* 
eentury, as Hidi and Haraekiewicz (2000, p. 151) have argued. Finding ways to stir up student 
interest in pursuing learning aetivities to maintain even the crudest indieator of engagement, 
simple sehool attendance, remains a challenge for most high-poverty seeondary schools. Prior 
research has noted that attendance (together with behavior and eourse grades) is much more 
important than test seores as a predietor of high sehool graduation (e.g., Balfanz, Herzog, & Mao 
Ivor, 2007; Allensworth & Easton, 2007). While test seores may beoome more important as 
states oontinue to institute graduation testing requirements, and oertainly predict the need for 
remediation as students transition to oollege, it is important not to ignore the impact of improving 
attendance on students’ oollege and oareer readiness. Attending olass in oollege and showing up 
to work every day are oritioal determinants of oollege and oareer suceess. Inoreasing attendanee 
in middle and high sehool is the first step to getting urban students ready for college and oareer. 
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Appendix B. Tables and Figures 

TABLE 1 

Differences in Group Means between Robotics Summer Program Students and Non-Program Students Before Propensity Score Matching, By Prior Grade Level 



Grade 5 


Program 

Non-program 

Std. Mean 

Students 

Students 

Diff 

(n=57 ) 

(n=4527) 



Grade 6 Grade 7 

Program Non-Program Std. Mean Program Non-Program 
Students Students Diff. Students Students 

(n=63) (n=4388) (n=46) (n=4406) 


Std. Mean 
Diff. 


Male 

0.72 (.45) 

0.48 (.50) 

0.53 

0.70 (.46) 

0.48 (.50) 

0.50 

0.76 (.43) 

0.49 (.50) 

0.00 

FRL 

0.86 (.35) 

0.89 (.32) 

-0.07 

0.87 (.34) 

0.88 (.32) 

-0.01 

0.80 (.40) 

0.86 (.34) 

-0.17 

Minority 

0.91 (.29) 

0.87 (.34) 

0.17 

0.97 (.18) 

0.88 (.33) 

0.53 

0.96 (.21) 

0.88 (.33) 

0.41 

Spec. Ed 

0.12 (.33) 

0.15 (.36) 

-0.10 

0.22 (.42) 

0.15 (.35) 

0.16 

0.20 (.40) 

0.14 (.35) 

0.17 

Overage 

0.11 (.45) 

0.21 (.41) 

-0.37 

0.27 (.45) 

0.21 (.41) 

0.15 

0.20 (.40) 

0.22 (.41) 

-0.08 

Changed schools 

0.02 (.13) 

0.07 (.25) 

-0.38 

0.06 (.25) 

0.08 (.27) 

-0.08 

0.04 (.21) 

0.07 (.25) 

-0.12 

Suspended 

0.07 (.26) 

0.08 (.27) 

0.03 

0.10 (.30) 

0.12 (.32) 

-0.09 

0.09 (.29) 

0.12 (.33) 

-0.15 

Summer School prior year 

0.46 (.50) 

0.23 (.42) 

0.46 

0.27 (.45) 

0.14 (.35) 

0.33 

0.52 (.51) 

0.12 (.33) 

0.84 

Prior Math z-score 

0.02 (1.13) -0.03 (.99) 

0.02 

0.35 (.82) 

0.38 (.92) 

0.01 

-0.12(1.01) 

0.06(1.0) 

-0.11 

Prior attendance 

96.9 (4.39) 95.0 (5.38) 

0.42 

95.96 (5.6) 

94.84 (5.74) 

0.18 

97.07 (2.56) 

94.35 (6.72) 

1.11 


Prior Y ear Characteristics of Students Post- Intervention School 


Enrollment 

232 (141) 

220 (135) 

0.07 

271 (146) 

221 (136) 

0.38 

300(161) 

226 (138) 

0.52 

%FRE 

81.2(13.8) 

84.9(12.5) 

-0.30 

82.3 (14.5) 

85.0(12.5) 

-0.21 

82.6(13.1) 

85.3 (11.9) 

-0.29 

Charter 

0.19 (.40) 

0.19 (.39) 

0.00 

0.13 (.34) 

0.18 (.39) 

-0.18 

0.07 (.25) 

0.17 (.38) 

-0.45 

Middle School 

0.28 (.45) 

0.25 (.43) 

0.06 

0.30 (.46) 

0.25 (.43) 

0.09 

0.22 (.42) 

0.27 (.44) 

-0.15 

Middle High 

0.16 (.37) 

0.13 (.34) 

0.06 

0.22 (.42) 

0.13 (.33) 

0.25 

0.22 (.42) 

0.13 (.34) 

0.18 

Avg. Math z-score 

0.04 (.51) 

0.02 (.49) 

0.05 

-0.05 (.46) 

0.03 (.48) 

-0.14 

0.05 (.46) 

0.02 (.49) 

0.14 


(Standard deviations in parentheses) 
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TABLE 2 

Results for Differences in Group Means between Robotics Summer Program Students and Comparison Students after Propensity Score Matching, By Prior Grade 
Level 

Grade 5 Grade 6 Grade 7 

Program Comparison Std. Mean Program Comparison Std. Mean Program Comparison Std. Mean 



Students 

(n=57) 

Students 

(n=169) 

Diff 

Students 

(n=63) 

Students 

(n=189) 

Diff 

Students Students 

(n=46) (n=128) 

Diff 

Male 

0.72 (.45) 

0.73 (.44) 

-0.03 

0.70 (.46) 

0.70 (.46) 

0.00 

0.76 (.43) 

0.76 (.43) 

0.00 

FRL 

0.86 (.35) 

0.89 (.32) 

-0.08 

0.87 (.34) 

0.89 (.31) 

-0.06 

0.80 (.40) 

0.80 (.40) 

0.01 

Minority 

0.91 (.29) 

0.92 (.27) 

-0.04 

0.97 (.18) 

0.97 (.16) 

-0.03 

0.96 (.21) 

0.96 (.19) 

-0.04 

Spec. Ed 

0.12 (.33) 

0.09 (.29) 

0.09 

0.22 (.42) 

0.18 (.39) 

0.10 

0.20 (.40) 

0.20 (.40) 

-0.02 

Overage 

0.11 (.45) 

0.12 (.33) 

-0.06 

0.27 (.45) 

0.25 (.43) 

0.05 

0.20 (.40) 

0.21 (.40) 

-0.04 

Changed schools 

0.02 (.13) 

0.02 (.13) 

0.00 

0.06 (.25) 

0.05 (.21) 

0.07 

0.04 (.21) 

0.06 (.24) 

-0.09 

Suspended 

0.07 (.26) 

0.07 (.26) 

0.00 

0.10 (.30) 

0.06 (.24) 

0.11 

0.09 (.29) 

0.12 (.32) 

-0.10 

Summer School prior year 

0.46 (.50) 

0.47 (.50) 

-0.03 

0.27 (.45) 

0.25 (.43) 

0.05 

0.52 (.51) 

0.46 (.50) 

0.13 

Prior Math z-score 

0.02 (1.13) 0.04 (1.10) 

-0.02 

0.35 (.82) 

0.37 (.80) 

-0.02 

-0.12(1.01) 

-0.12 (.93) 

0.00 

Prior attendance 

96.9 (4.39) 96.8 (4.1) 

0.02 

95.96 (5.6) 

96.39 (4.77) 

-0.08 

97.07 (2.56) 

97.50 (2.39) 

-0.17 

Prior Y ear Characteristics of Students Post- Intervention School 







Enrollment 

232 (141) 

245 (164) 

-0.09 

271(146) 

274(161) 

-0.02 

300(161) 

310(157.1) 

-0.06 

%FRE 

81.2(13.8) 

81.6(17.8) 

-0.03 

82.3 (14.5) 

82.0(16.1) 

0.02 

82.6(13.1) 

83.6(14.9) 

-0.06 

Charter 

0.19 (.40) 

0.19 (.39) 

0.03 

0.13 (.34) 

0.17 (.38) 

-0.14 

0.07 (.25) 

0.14 (.35) 

-0.30 

Middle School 

0.28 (.45) 

0.23 (.42) 

0.10 

0.30 (.46) 

0.34 (.48) 

-0.09 

0.22 (.42) 

0.31 (.46) 

-0.22 

Middle High 

0.16 (.37) 

0.12 (.32) 

0.11 

0.22 (.42) 

0.19 (.39) 

0.07 

0.22 (.42) 

0.24 (.43) 

-0.06 

Avg. Math z-score 

0.04 (.51) 

0.07 (.53) 

-0.06 

-0.05 (.46) 

-0.03 (.53) 

-0.02 

0.05 (.46) 

-0.06 (.54) 

0.21 


(Standard deviations in parentheses) 
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TABLE 3 

Results from HEM Models Predicting Baseline Student Eevel Attendance Rates (Year Prior to Intervention) for Pull Sample and for Sub-Sample of 
Eow-Performing Students in Mathematics 

Pull Sample Eow-Performing Students 

(n=652) (n=227) 



Coefficient 

Std. Error 

Coefficient 

Std. Error 

Intercept 

96.84*** 

(0.65) 

95.86*** 

(0.95) 

Treatment 

-0.31 

(0.76) 

-0.42 

(1.23) 

Sixth grade prior year 

-0.58 

(0.38) 

-0.61 

(0.88) 

Seventh grade prior year 

0.55 

(0.42) 

1.72 

(0.91) 


*p<.05 **P<.01 ***P<.001 

Grade level dummy variables were grand-mean centered. 
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TABLE 4 

Results from HEM Models Predieting Student Eevel Attendanee Rates in the Year Eollowing Intervention for Eull Sample and for Sub-Sample of 
Eow-Performing Students in Mathematies 


Pull Sample 
(n=652) 

Coefficient Std. Error 


Eow-Performing Students 
(n=227) 

Coefficient Std. Error 


Intercept 

95.61*** 

(.20) 

93.82*** 

(0.46) 

Treatment 

1.38** 

(.39) 

2.58** 

(0.90) 

Male 

0.13 

(.39) 

0.18 

(1.04) 

ERE 

-0.71 

(.54) 

-0.87 

(1.28) 

Minority 

-0.26 

(.83) 

-0.78 

(2.25) 

Special Ed 

-0.46 

(.48) 

-0.53 

(0.87) 

Overage for grade 

-0.03 

(.45) 

0.98 

(0.86) 

Changed schools 

-0.15 

(.87) 

-1.27 

(1.89) 

Suspended 

-0.82 

(.65) 

-2.01 

(1.41) 

Summer school prior year 

0.92* 

(.36) 

1.12 

(0.85) 

Prior attendance 

0.65*** 

(.05) 

Q 

(0.08) 

Prior Math z-score 

0.75** 

(.21) 

0.52 

(0.86) 

Sixth grade prior year 

-0.13 

(.42) 

0.31 

(1.31) 

Seventh grade prior year 

0.64 

(.46) 

2.12 

(1.14) 

Prior Year Characteristics of Students’ Post-Intervention School 



Enrollment 

0.00 

(.00) 

-0.01 

(0.00) 

%PRL 

0.01 

(.02) 

0.03 

(0.05) 

Charter 

0.05 

(.56) 

-0.51 

(1.38) 

Middle School' 

0.93* 

(.46) 

1.62 

(1.02) 

Middle High 

1.45** 

(.55) 

1.96 

(1.20) 

Avg. Math z-score 

-.0.43 

(.59) 

0.40 

(1.47) 

Avg. Attendance 

0.10 

(.07) 

0.10 

(0.14) 


*p<.05 **P<.01 ***P<.001 


* Reference groups are Grade 5 in prior year, K8 schools, did not change schools during prior year, etc. All covariates are grand-mean centered. 
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